[orchestrator-v2] fix(orchestrator): Restore Claude session continuity for resume, wake, and idle release#3860
Conversation
Introduce a provider-agnostic continuation request queue, worker, and optional hasPendingBackgroundWork idle pin so adapters can surface post-settle native wake traffic as first-class runs. No adapter offers continuations yet; behavior is unchanged until Claude or Grok specialty commits wire attach mode.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
vouch:unvouched
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| ordinal: projection.messages.length + 1, | ||
| }); | ||
| const commandId = CommandId.make(`provider-continuation:${messageId}`); | ||
| yield* threads.dispatch({ |
There was a problem hiding this comment.
🟡 Medium orchestration-v2/ProviderContinuationService.ts:42
When a provider continuation request is dispatched, the worker sends a message.dispatch using only request.threadId and ignores request.providerThreadId. The orchestrator resolves the target provider thread from the thread's current activeProviderThreadId, so if the active provider thread changed between when the wake was queued and when it is drained, the continuation run starts against the wrong provider thread. The buffered Claude wake messages pinned to the original native thread are never consumed by that run, so the background-task wake is silently lost until some later turn happens to target the original thread. Consider including request.providerThreadId (or the original native thread identifier) in the dispatch so the continuation is routed to the correct provider thread regardless of any subsequent thread switch.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/orchestration-v2/ProviderContinuationService.ts around line 42:
When a provider continuation request is dispatched, the worker sends a `message.dispatch` using only `request.threadId` and ignores `request.providerThreadId`. The orchestrator resolves the target provider thread from the thread's *current* `activeProviderThreadId`, so if the active provider thread changed between when the wake was queued and when it is drained, the continuation run starts against the wrong provider thread. The buffered Claude wake messages pinned to the original native thread are never consumed by that run, so the background-task wake is silently lost until some later turn happens to target the original thread. Consider including `request.providerThreadId` (or the original native thread identifier) in the dispatch so the continuation is routed to the correct provider thread regardless of any subsequent thread switch.
| @@ -3643,6 +3861,7 @@ const makeDefaultClaudeAdapterV2 = Effect.fn("ClaudeAdapterV2.layer")(function* | |||
| const idAllocator = yield* IdAllocatorV2; | |||
| const queryRunner = yield* ClaudeAgentSdkQueryRunner; | |||
There was a problem hiding this comment.
🟡 Medium Adapters/ClaudeAdapterV2.ts:3862
makeDefaultClaudeAdapterV2 yields ProviderContinuationRequests and passes it into the adapter, but the exported layer omits ProviderContinuationRequests from its requirement type. Since ProviderContinuationRequests is a Context.Reference, composing ClaudeAdapterV2.layer with ProviderContinuationRequests.layer resolves the default { offer: () => Effect.void } inside the adapter, silently dropping every continuation request. Claude background-task wake turns are never enqueued, breaking session continuity. Add ProviderContinuationRequests to the Layer.Layer requirement type so callers must supply (or inherit) the live queue.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts around line 3862:
`makeDefaultClaudeAdapterV2` yields `ProviderContinuationRequests` and passes it into the adapter, but the exported `layer` omits `ProviderContinuationRequests` from its requirement type. Since `ProviderContinuationRequests` is a `Context.Reference`, composing `ClaudeAdapterV2.layer` with `ProviderContinuationRequests.layer` resolves the default `{ offer: () => Effect.void }` inside the adapter, silently dropping every continuation request. Claude background-task wake turns are never enqueued, breaking session continuity. Add `ProviderContinuationRequests` to the `Layer.Layer` requirement type so callers must supply (or inherit) the live queue.
github-actions
Bot
added
size:XL
| providerSessionId: input.providerSessionId, | ||
| pinnedForMs: now - pinnedSinceMs, | ||
| }); | ||
| yield* Ref.update(sessions, (latest) => { |
There was a problem hiding this comment.
🟡 Medium orchestration-v2/ProviderSessionManager.ts:622
The deferral update in releaseIfStillIdle does not revalidate idleGeneration, so if a turn starts and completes while hasPendingBackgroundWork is yielding, this stale callback overwrites the pinnedSinceMs reset done by markBusy with the old value and re-arms the timer. The pin cap is then measured across the intervening active turn, causing a background task from the new idle period to be released earlier than maxIdlePinMs. Add a check that latestEntry.idleGeneration === input.generation before applying the deferral update, matching the guard already used for the release branch.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/orchestration-v2/ProviderSessionManager.ts around line 622:
The deferral update in `releaseIfStillIdle` does not revalidate `idleGeneration`, so if a turn starts and completes while `hasPendingBackgroundWork` is yielding, this stale callback overwrites the `pinnedSinceMs` reset done by `markBusy` with the old value and re-arms the timer. The pin cap is then measured across the intervening active turn, causing a background task from the new idle period to be released earlier than `maxIdlePinMs`. Add a check that `latestEntry.idleGeneration === input.generation` before applying the deferral update, matching the guard already used for the release branch.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review 3 blocking correctness issues found. This PR introduces significant new runtime behavior for Claude session continuity, including background task tracking, wake turns, continuation requests, and modified idle release logic. Four unresolved review comments exist, including a high-severity issue about CAS reject still emitting reopen events. You can customize Macroscope's approvability policy. Learn more. |
…, and idle release Resume persisted Claude sessions after provider session recycle. Surface background wake turns as continuation runs: when a background task (run_in_background Bash) settles, the Claude CLI streams a whole new turn over the still-open SDK query; buffer null-activeTurn wake messages, request one internal continuation run per wake, and drain the buffer in attach mode without re-prompting the CLI, reporting pending background work so idle release stays pinned until the wake is ingested. Prevent session release from hanging on idle CLI reads. Require claude-agent-sdk 0.3.205 and align the replay testkit with it: derive requestId from toolUseID for replayed permission requests and fail the replay when canUseTool returns null. Regenerate the native-stack patch header so current aube parses it.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit c489f02. Configure here.
| return current; | ||
| } | ||
| return new Map(current).set(input.taskId, subagent); | ||
| }); |
There was a problem hiding this comment.
CAS reject still emits reopen
High Severity
The session-registry CAS can refuse a resume reopen when a newer terminal entry won the race, but turn maps are already overwritten and subagent.updated / node.updated still emit the reopened running state with a cleared result. Projection can finish on a stale running row after the task actually completed, which is the clobber the CAS was meant to prevent.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit c489f02. Configure here.
1 participant
Summary
Mirrors the shape of the Grok v2 PR (#3578): one shared provider-continuation
plumbing commit (identical to the one carried by #3578) plus one Claude
specialty commit.
Four related failure modes around Claude provider session lifetime and
background work are fixed:
Session ID ... is already in use);the adapter now resumes the persisted native session instead of recreating
it.
run_in_backgroundBash) vanished when they settled; theCLI's post-settle wake turn is now ingested as a first-class continuation
run, and idle release stays pinned while background work is pending.
run_in_background) lost theircompletion when it arrived after the root turn settled, and resuming a
completed subagent via SendMessage was invisible: the row kept its stale
first result forever. Both lifecycles now project correctly across turns
and runs.
message read, orphaning the CLI child; the release path now unblocks the
pending read and persists released events even if a finalizer wedges.
Branch shape
t3code/codex-turn-mappingfeat(orchestrator): Add shared provider continuation plumbing(C0, the same commit object carried by [orchestrator-v2] fix(orchestrator): Harden Grok v2 settlement and steer message visibility #3578; not a standalone PR)
fix(orchestrator): Restore Claude session continuity for resume, wake, and idle release(this PR's reviewable delta)Problem and Fix
ProviderSessionManageridle-releases a provider session (30 min default), the adapter's in-memoryopenedNativeThreadsset is lost. The next turn'sopenQueryopens the native thread with{sessionId}(create semantics,--session-id) instead of{resume}, and the Claude CLI fails fast because the session id already exists on disk.openQuery, treat a persisted provider turn as proof the native session exists:providerTurnOrdinal > 1forces{resume}alongside the existing in-memory and rollback signals.ClaudeAdapterV2.handleSdkMessageearly-returned whenactiveTurnwas null, so every message of the wake turn was dropped.activeTurnwake evidence now buffers per native thread. A pending-task notification or result offers one continuation request (deduped per wake) to a newProviderContinuationRequestsqueue; theProviderContinuationServiceworker dispatches an internalmessage.dispatch(createdBy: "agent",creationSource: "provider",queue_after_active) so the wake flows through the normal run pipeline. The continuation turn sends no prompt to the CLI and drains the buffered wake messages into itself.task_notificationarriving after settle hit the same null-activeTurndrop, and even when replayed, the continuation turn's fresh per-turn maps could not find the subagent row created by the earlier turn.sessionSubagentsByTaskId) lets a wake-replay turn hydrate and terminalize a subagent created by an earlier, settled turn.task_startedwith the same task id but the SendMessage call'stool_use_id, then a secondtask_notificationcarries the new answer. Subagent status was monotone, so the re-open was ignored; the SendMessage tool_result (a delivery ACK) could also terminalize the re-opened row with ACK JSON as its "result".task_startedfor a known task id is an authoritative CLI lifecycle event and may re-open a terminal row (stale result and progress cleared); a latetask_progressstill cannot. Only a tool_result whose id is absent fromtoolCallsmay terminalize a subagent, which excludes SendMessage ACKs (Agent launches never entertoolCalls; they project as subagent rows). Post-settle resume frames pre-open the session registry entry so idle stays pinned and the eventual notification counts as wake evidence.RunExecutionService.routeProviderEventonly persists parent-thread subagent events whoserunIdmatches the currently executing run, and a run's ingestion fiber only lingers past settle while it has active child subagents. A subagent that completed in-turn and was resumed later emitted events attributed to the launch run, whose fiber was already gone; they were silently dropped and the row froze on the stale result.runIdto the run whose turn processes the resume. The continuation run's event filter then accepts the events, and its child-lifecycle tracking keeps its ingestion fiber alive until the resumed task completes. Parenting (parentNodeId/rootNodeId) stays with the launch run's root node.hasPendingBackgroundWorkruntime capability;releaseIfStillIdledefers theidle_timeoutrelease while it reports true, re-arming the timer, with cumulative deferral capped bymaxIdlePinMs(default 4h).closeSessionfinalizer. Interruption callsreturn()on the SDK's rawsdkMessagesgenerator, which queues behind the in-flight idle read forever; the scope close never completes andquery.closeis never reached, orphaning the CLI child and leaving the projection stuck on"ready".claudeQueryMessageshands the stream an iterable whose[Symbol.asyncIterator]()returns theQueryitself. The SDK'sQuery.return()runscleanup()first, closing the transport and terminating the CLI child, so interruption completes and the remaining finalizers run.requestIdoncanUseTooloptions and allows a nullPermissionResult.@anthropic-ai/claude-agent-sdkfloor to^0.3.205(lockfile resolves 0.3.205), deriverequestIdfromtoolUseIDfor replayed permission requests, and fail the replay ifcanUseToolreturns null. The native-stack patch file also gains itsdiff --githeader so current aube can parse it.Defensive Fixes
releaseEntryforever with no released events and no log line.releaseEntryforks the scope close and joins with a 30s timeout. On timeout it logsprovider-session-scope-close-timeout, attaches a detached observer for late completion or failure, and persists the released events regardless.hasPendingBackgroundWorkyields to the adapter between the idle check and the release, so a release could race a session that turned busy during the check.releaseEntrytakesonlyIfIdleGenerationand revalidates busyCount/idleGeneration inside its atomic entry removal.Validation
vp check: passpnpm install --frozen-lockfile: pass after the SDK bump to 0.3.205vp run typecheck: pass (full monorepo)vp test .../ClaudeAdapterV2.test.ts: 24/24, including background waketurns (buffer + single deduped continuation request; drain into a
continuation turn with no CLI prompt; racing user turn leaving the buffer
for the queued continuation; null-summary notification clearing the
pending-task registry; spurious continuation settling immediately) and the
subagent lifecycle (post-settle completion via the session registry;
in-turn SendMessage resume with re-open, ACK guard, and run
re-attribution; post-settle resume whose frames race past settle; late
task_progressunable to re-open a terminal row)vp test .../ProviderSessionManager.test.ts: 18/18 (idle release deferredwhile background work is pending, pinned session released once
maxIdlePinMsexpires, turn starting during the pending-work check neverlosing the session)
ClaudeReplayFixtures.integration.test.tsandOrchestratorReplayFixtures.integration.test.ts(claude fixtures): pass(in-turn background bash, background subagent, monitor) and
claude-continuation (post-settle bash wake, post-settle subagent
completion, SendMessage resume of a completed subagent), all passing,
covering both resume shapes (in-turn nudge with lingering launch-run
ingestion, and CLI autonomous-turn nudge requiring the run
re-attribution)
audits of run, subagent, and message projections in sqlite
all Medium findings addressed. The subagent resume and re-attribution
deltas were reviewed by grok-4.5 across four rounds; final verdict
approve, no High or Medium findings
Known limitations
the server and the wake is lost (out of scope).
active turn, exactly as today; this PR only changes the previously-dropped
null-
activeTurnpath.today and that wake is lost (same outcome as before, minus up to 4 hours).
failure) is logged but not retried; the wake buffer then sits until a later
turn drains it or the pin cap releases the session.
cleanup;
provider-session-scope-close-timeoutis the operational signal.optional attempt-metadata link for that row can resolve to none (its
runIdmoves while its parent node stays with the launch run). Thetimeline entry itself still renders.
Note
Restore Claude session continuity for resume, wake, and idle release in orchestrator-v2
ProviderContinuationRequests, a context-level queue that the Claude adapter uses to signal background work completions, andProviderContinuationServicewhich drains that queue and dispatches internal messages to resume sessions.makeClaudeAdapterV2to buffer SDK messages that arrive outside an active turn (wake messages), offer a single continuation request per wake, and drain buffered messages into agent-originated continuation turns instead of sending a new user prompt.hasPendingBackgroundWorktoProviderAdapterV2SessionRuntime, allowingProviderSessionManagerV2to defer idle release while background tasks, running subagents, or buffered wake messages exist, with a configurable cap (default 4 hours).ProviderSessionManagerV2to persist release and notify subscribers even when adapter scope close hangs (30s timeout), and to guard releases against stale idle-generation decisions.claudeQueryMessagesiterator to callQuery.return()on stream interruption, running cleanup and closing the transport to avoid deadlocks during idle interrupts.@anthropic-ai/claude-agent-sdkfrom^0.3.170to^0.3.205.Macroscope summarized c489f02.